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Abstract 

Distributed optimization is a fundamental mathematical theory for parallel and distributed systems. 
Several applications are normally designed based on such a theory, where parties cooperatively exchange 
messages with little or no central coordination to achieve some goals. In many situations, the transactions 
among the parties must be private, such as among members of social networks, hospitals, companies 
in a free market, banks, and state governments, to mention a few. Existing privacy preserving solution 
methods for optimization problems are mostly based on cryptographic procedures and thus have the 
drawback substantial computational complexity, which is infeasible for large scale networks. The 
availability of distributed optimization solution methods that are private per se are therefore highly 
desirable and sometimes the only ones viable. Surprisingly, little attention has been devoted thus 
far to the development of a general theory for such privacy preserving distributed optimization. In 
this survey paper, a new general framework of existing transformation based mechanisms for privacy 
preserving distributed optimization is presented. The privacy preserving properties that are inherent in 
the classical decomposition techniques, such as primal decomposition, dual decomposition and state- 
of-the-art methods, such as alternating direction method of multipliers are investigated. A number of 
examples is provided to illustrate the need of a new theory of per-se privacy preserving optimization. It 
is concluded that the theory is still in its infancy and that huge benefits can be achieved by a substantial 
development. 
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I. Introduction 

Several real-world optimization problems involve parties or nodes interacting via some net- 
works that must collaborate to solve an optimization problem for mutual benefit [[Tl, |l2l. For 
example, in the business sector, independent companies need to interact for completing a common 
business and thus have to work together to optimize their joint operations. Normally, optimization 
solvers require much public data sharing among the parties, which may substantially hinder or 
prevent the cooperation for optimization due to privacy concerns. The fundamental question 
is how to solve optimization problems among parties that would receive much benefit by 
collaboration and yet are unwilling to share their data without preserving privacy. 

Cryptography is the standard approach to preserve privacy in distributed optimization solvers jSl. 
However, cryptography may introduce substantial overhead among the nodes due to the exchange 
of security information and coordination. Cryptography is prone to attack by third parties 
who may inadvertently own the cryptographic keys. Moreover, for large networks, namely 
for networks with a large number of parties, it might be prohibitive if not impossible to use 
cryptography. Therefore, it is highly desirable to have distributed solution algorithms without 
introducing extra coordination or overhead and that naturally preserve privacy in the transactions. 

The availability of these distributed solution algorithms posses many appealing merits, which 
are desirable in practice, e.g., efficiency, scalability, natural (geographical) distribution of prob- 
lem data. More importantly, they are per-se privacy preserving without requiring any extra 
coordination or overhead. In this paper, we propose a systematic study of methods ensuring 
both the efficient solution of distributed optimization problems and the privacy of the associate 
transactions. We further motivate the need of such a theory by some simple illustrative examples. 

A. Motivating examples 

Here, we present some real-world examples that motivate and highlight the strong need of a 
per se privacy preserving optimization theory. 

Example 1 (Privacy preserving data mining l^): Suppose that there are two different government agen- 
cies, a local Agency 1 and a nation-wide Agency 2 that store their information at their databases. The general 
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goal is that both agencies have to collaborate and provide a joint classification of possible security threats. The 
main difficulty in this collaborative procedure is that several lower clearance level requirements at Agency 1 
prevents Agency 2 from granting Agency 1 open access to the data base of Agency 2. Moreover, even if 
Agency 1 can gain temporary access to the database of Agency 2, the databases of these agencies are restricted 
to their current physical locations due to security policies. This is a typical example of distributed data mining, 
which relies entirely on distributed optimization. The example can be easily generalized to many agencies. This 
examples can be solved by using the per-se privacy preserving methods proposed in § |Vl 



Example 2 (Production line problem fiSj): We assume that Company 1 produces and sells some products 
(e.g., bags, etc) with various quality. These products can provide different profit levels to the company as well. 
The general problem is to determine how many pieces of each kind of product must the company produce in 
order to maximize the profit, assuming that all the made products will be bought. At some point. Company 1 
would like to increase its profit and, therefore, outsources one or more production stages to another company. 
Company 2. However, Company 2 is not willing to disclose its capacity or its production times to Company 1. 
The only thing that Company 2 would like to share is the charging cost of each production stage. How can 
Company 1 decide if the deal is worthwhile based on this limited information? This is case of production line 
distributed optimization, which can be easily generalized to many companies. This examples can be solved in 
a per-se privacy preserving manner by using methods in § IIVIVIIIII 

These real-world problems could be solved by applying cryptography, as proposed in literature. 
However, a per-se preserving optimization approach can be instead used with much more 
efficiency, as argued in this paper. In the following, we briefly present some representative 
approaches and highlight our original contribution. 

B. Background and Contribution 

Existing privacy preserving methods used in distributed optimization solvers assume that nodes 
or parties of a network own some constraints and the overall network wants to optimize a cost 
function that may be given by the composition of functions belonging to the nodes. Note that in 
general, the network cost function may be given by a separable composition of the nodes' s cost 
functions, or by a non-separable composition. The classic distributed optimization problem solver 
consists in running at the nodes algorithms that transmit to other nodes or network coordinator, 
data, such as decision variables, and receive by the network or by the coordinator back further 
information to iterate and reach eventually the computation of the optimal solution |[T1. There are 
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essentially two approaches: cryptographic methods, and non cryptographic methods. The second 
approach, can then be further divided into algebraic transformation-based methods, which we 
survey and generalize in this paper, and potential decomposition based approaches, which we 
investigate substantially in this paper. 

Cryptographic primitives include secure multiparty computations [|6]|, pseudo random generator 
0, [HI, homomorphic encryption flU, [fTOll . [fTT]|. These methods use cryptographic tools such 
as zero-knowledge [12], oblivious transfer [fT3l . 1-out-of-n oblivious transfer [[T4l|. oblivious 
evaluation of polynomials [|T5l . secret sharing [fT6ll . and threshold cryptography ifTTl . In general, 
the area of cryptography -based privacy preserving optimization is very well investigated [3J, 
[fTSl — [[2T | . In the context of optimization problems, cryptographic tools are used for securely 
perform iterations of well known simplex algorithm and interior-point algorithm so that the 
sensitive data is not disclosed during the methods, see [f22 | — [|24 || for secure simplex variants 
and [5] for secure variants of interior-point method. In terms of security, those cryptographic 
methods are highly desirable, though they are highly unfavorable in terms of computational 
complexity and efficiency 

Developing solution methods based on algebraic transformations that preserve privacy of data 
without cryptography has attracted the interest of the research community Hl—lO, ll25l — OTI. 
We refer to those approaches as transformation methods in general. The key idea of these is 
to use algebraic manipulations to disguise the original problem into an equivalent problem so 
that the private data of each node is hidden. However, in these interesting papers, only some 
specific problems have been considered and there is not any attempt to establish a systematic 
approach. We strongly believe that the transformation based privacy preserving approaches still 
need to be systematically investigated. Moreover, decomposition methods [[T]|, [f32ll . and state- 
of-the-art distributed optimization methods, such as alternating direction method of multipliers 
(ADMM) [|33l are less investigated in the context of privacy preserving problem solution methods. 
However, these approaches posses inherent privacy preserving properties, and therefore have a 
huge potential to handle privacy issues in a more efficient manner. 

Given the drawbacks of cryptographic approaches, investigating efficient and scalable methods 
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that are per-se privacy preserving is of crucial importance from a theoretical, as well as from 
a practical perspective. Therefore, in this paper, we place a greater emphasis on transformation 
methods and optimization approaches, such as decomposition methods and ADMM methods 
that posses inherent privacy preserving characteristics, as opposed to treatments based on well 
investigated cryptographic primitives. Of course, there are many papers that describe in details 
cryptographic methods and are extraneous to the main focus of this paper (see, e.g., [3]). In par- 
ticular, we summarize the existing privacy preserving solution methods based on transformation 
approaches by a more generic canonical form. Then, we discuss in detail a general decomposition 
structure together with classical decomposition techniques, including primal decomposition and 
dual decomposition, as well as state-of-the-art method ADMM. Their inherent privacy preserving 
properties are investigated. The importance of these per-se privacy preserving methods is high- 
lighted through a number of examples and their performance is compared in terms of efficiency, 
scalability, complexity and many others. To the best of our knowledge, this is the first paper 
that provides a general classification and comparison of per-se privacy-preserving methods in 
optimization, and introduces the inherent privacy properties of decomposition methods. 

The rest of the paper is organized as follows. In §|II]we present some basic definitions that are 
useful for describing the properties of privacy preserving optimization. A general formulation to 
model the existing privacy preserving techniques is presented in § Unl Optimization approaches, 
such as decomposition, ADMM methods in the context of privacy preserving optimization are 
extensively discussed in § UlI] and |Vl In § |VIl we provide a summary, including comparisons of 
different methods for privacy preserving and future directions. Conclusions are given in § IVIII 

Notations: Boldface lower case and upper case letters represent vectors and matrices respec- 
tively and calligraphy letters represent sets. The set of real n-vectors is denoted R", and the set 
of real m x n matrices is denoted IR™^". The identity matrix is denoted by I. The superscript 

T 

(■) Stands for transpose. Vectors and matrices are delimited with square brackets, with the 
components separated by space. The ith submatrix of a matrix is usually denoted by using a 
subscript. We use parentheses to construct column vectors from comma separated lists, e.g., 
(a, b, c) = [a^ c^]^. We denote by ||x||i the £i-norm and by ||x||2 £2-norm of the vector x. 
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II. Per-se privacy preserving definitions 

Many standard privacy/security conventional definitions are already adopted in cryptographic 
literature; see for example (SI, [l34l — [l38l . However, such definitions are not directly applied or 
adopted in the context of transformation based and decomposition based optimization methods. 
This is because the key mechanisms used for preserving the privacy in cryptographic protocols 
are entirely different from those that are to be used in optimization based approaches. Theoretical 
foundations for defining the privacy of optimization methods deserve attention in its own right. 
However, to highlight the appealing privacy preserving properties associated with optimization 
based approaches, and to provide a cohesive discussion, we give here some basic definitions. 

Definition 1 (Optimization problem): We consider the following standard notation to describe 

the problem of finding a point x that minimizes the function /o(x) subject to a number of 

inequality and equality constraints: 

minimize /o(x) 

subject to /i(x) < 0, i = 1, . . . , g (1) 

hi{x.) =0, i = l,...,p . 
We call ([U), an optimization problem. Here /o : R" — )■ IR is called the objective function, /i(x) < 

0, 2 = 1, . . . , g are called inequality constraints with the associated inequality constraint functions 

fi : R" — )■ R, i = 1, . . . ,q, hi{x.) = 0, i = 1, . . . ,p are called equality constraints with the 

associated equality constraint functions hi : R" — )■ R, i = 1, . . . ,p, and x = (xi, . . . , x„) G R" 

is called the optimization variable [f39ll . The terms optimization variable, decision variable, and 

primal variable are all synonymous. 

Definition 2 (convex problem): A convex optimization problem is a one of the same form 

as (dl) provided some assumptions hold. In particular, the convex problem is 

minimize /o(x) 

subject to /j(x) < 0, z = 1, . . . , g (2) 
Cx - d = , 

where the functions fi, z = 0, . . . , g are convex and hi, i =, . . . ,p are affine, i.e., the equality 
constraint functions are given by C G R^^" with rank(C) = p and d G R'''. The optimization 
variable is x = {xi, . . . ,Xn) G R". Typically, we have p < n in practice, otherwise the only 
potential solution is CM, where is called the pseudo-inverse of C (see lf39l § A. 5. 4]). 
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Definition 3 (K -party environment): A set of K parties is called an A^-party environment. 

Through out this paper, we consider problems of the form ([T])— (|2l), which are to be solved 
in an i^-party environment via coordination among the parties. The global variable x, objective 
function /o, inequality functions /, i = 1, . . . ,q, equality functions hi i = 1, . . . , m, different 
subset of components of x, different subset of functions, and/or function definitions themselves 
can be spread out among K parties. We usually use the terms ownership, private data to indicate 
clearly spreading among K parties such functions, data, etc. Note that the terms party and entity 
are often used interchangeably in this paper. 

Examples.- Consider the case /o(x) = J2iLi^J^i^ where € ]R"% G H"' with = 
and X = (xi, . . . ,xk)- Here, the objective function /o and the optimization variable x is spread out among 
ii' -parties, such that and Xj are owned by party i. 

Definition 4 (Semi-honest adversary): In a multi-party environment, an entity involved in 
solving a global optimization problem of the form ([U), centrally or in a coordinated fashion 
with other entities, is called a semi-honest adversary, if it executes its prescribed computations 
correctly, but keeps a record of all information it receives from other parties and tries to learn 
properties of others' private data. 

The definition above is inspired form [|40ll and is similar to the good and the passive adversary 
models given in pT| and [|42l. respectively. In a multi-party environment, there can be more 
than one adversaries. We consider the semi-honest adversary model throughout this paper. 

Definition 5 (Inputs and outputs of a convex problem): Consider the convex problem We 
call the set of problem parameters, the terms C,d, those ones required to define the functions 
/o, fi and/or C, d together with the functions themselves as inputs of problem Moreover, 
we call the solution and the optimal value of problem ^ as outputs. 

Example 4: Consider the following linear program: 

minimize c^x 

subject to Ax < b (3) 
X > , 

where the variable is x £ M" and problem data are c G M", A e M™^", and b £ M"*. Suppose x* solves the 
problem. The parameters {c, A, b} are the inputs of the problem and {x*,p*} are the outputs of the problem. 
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where p* = c^x*. 

Definition 6 (Input privacy): We say that the mechanism for solving an optimization problem 
yields input privacy for a subset C of the input if during the solution method stages, only a 
transformed variant of the original objects in C is available to the adversary and such that 
recovering the original C is impossible without the knowledge of the transformation. Moreover, 
we say that the solution method for an optimization problem yields input privacy for C also if 
C is unknown to the adversary. 

Example 5: Consider once again the Hnear program (O, which is to be solved by Ahce and Bob. The input 
ownership is as follows: Alice owns c and Bob owns {A,b}. Now suppose they use the following solution 
procedure: instead of the input c, Alice uses c, where c = ac and a is a positive scalar known by Alice only. 
Similarly, instead of the input {A, b}. Bob uses {A, b}, where A = /3A, b — /3b, and /3 is a positive scalar 
known by Bob only. The result is the following optimization problem: 

minimize c^x 

subject to Ax < b (4) 
X > , 

where the variable is x. Problem ^ and (|4]i are clearly equivalent, because Ax < b <^ Ax < b and 
minimizing c^x is identical to minimizing c^x — ac^x with a > 0. Either Alice or Bob can use any linear 
programming solver for solving the problem. This solution procedure yields input privacy for Alice's data c 
and for Bob's data {A,b} because without knowing /3, Alice cannot recover Bob's data {A, b}, by using 
{A, b}., and in addition, without knowing a. Bob cannot recover Alice's data c, by using c. 

Definition 7 (Output Privacy): We say that the mechanism for solving an optimization prob- 
lem yields output privacy for a subset X of the output if at the end, only a transformed variant 
of X is available to the adversary such that recovering the original X is impossible without 
knowing the transformation. Moreover, we say that the solution method for an optimization 
problem yields output privacy also if X is unknown to the adversary at the end. 

More examples will be discussed in § UlI] and § |lVl We emphasize that investigating a 
comprehensive treatment of a set of mathematical definitions is out of the scope of this paper. We 
believe that the basic definitions dl])-© given above are sufficient for surveying the appealing 
aspects and properties of privacy preserving optimization approaches presented in this paper. 
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III. Transformation Based Methods for Privacy Preserving 

In this section, existing transformation based methods proposed for privacy preserving linear 
programming [5, § 6-7] [|26l, [|27l, [H [|30l § 4] [|2l, 01, as well as nonlinear 

programming ll45l . Il46l are considered. The number of parties involved in a problem can be 
either two S, (23, [[301, [[311, [il, [gl or more [l26l--|l2l, [|45l, 061. In the sequel, we provide 
a generic description for such privacy preserving solution methods proposed in the literature. 

Transformation method is directly based on the notion of equivalence of optimization prob- 
lems [|39l § 4.1.3]. Two problems are called to be equivalent if, from a solution of one, a solution 
of the other is readily derived. There are many general transformations that yield equivalent 
problems. We next propose two transformations, which captures all the problem formulations 

in m, ina-im, m, uni, ma-gii. 

A. Transformation via Change of Variable 

Suppose the original problem to be solved in a privacy preserving manner is given by We 
denote by V the set of points for which the objective and all constraint functions are defined, or 
domain of problem ©, i.e., V = HLodom fi n IR" Q. Let : R'" ^ IR" be a function, with 
image covering the problem domain V. Now we perform the following change of variables: 

X = 0(z) . (5) 

The resulting problem is given by 

minimize /o(0(z)) 

subject to /j(0(z)) < 0, i = 1, . . . , g (6) 
C0(z) - d = , 

where the variables are z G R"^. It follows that if x solves problem then z = 0^^(x) solves 
problem Moreover, if z solves problem then x = 0(z) solves problem Thus, the 
two problems are equivalent. Such a transformation is used to ensure privacy, as we see next. 

Privacy Properties 

If the function (p is chosen appropriately, then any solution method applied to the transformed 
problem (via change of variables) can yield the input privacy for many inputs (except for d) via 

*For many considered problem formulations in this paper, the domain V — M". 
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the function compositions 

/i(z) = /i(0(z)) , dom /i = {z G dom | 0(z) G dom fi} , i = 0, . . . ,q , 
hi{z) = C0(z) - d , dom = {z G dom | 0(z) G IR"} . 

The output privacy for the optimal solution is attained by the definition of (see ([5])). In the 
sequel, we highlight these privacy properties by using some examples, first, lets see some choices 
for the function 0. 

Example 6: 

• Scaling: Here, we simply use the change of variable x = (/)(z) = az, where a is a scalar. 

• Translation: Here, we use the following change of variable, x = 0(z) = z + a, where a G H". 

• Affine transformation: This is a generalization of both scaling and translation. Specifically, we use 
the following change of variable, x = 0(z) = Bz + a, where B £ H"^™ is a full rank matrix with 
rank(i?) ~ n and a £ H". Thus, a particular inverse transformation : H" — > R™ is given by 

z = cj)-^{^) = Btx-B^a , 
where B^^ = B^(BB^)~^, which is typically known as pseudo-inverse or Moore-Penrose inverse. 

• Nonlinear transformations: One example is as follows, Xi = (piizi) — exp (z^), where a; > 0. 

We see that all the approaches (H, [l25l--[l271, JSOj, |l3ll, Il43l--|l46l have used change of 
variables (affine transformations) as one of the mechanism for privacy preserving in their 
proposed solution methods. To illustrate this, we consider few key examples from the literature. 

Example 7 ( Computation outsourcing in the cloud, a 2-party environment H44\I ): Suppose a cloud customer 
wants to outsource to the cloud his linear program 

minimize c^x 

subject to Ax = b (7) 
Bx > , 

where the variable is x £ M" and the problem data are c £ M", A £ ]R"'^" with m < n, nonsingular 
B £ M"^", and b £ H™. The customer does not want to reveal problem data c, A, b, B and the solution x* 
of the problem to the cloud, i.e., input privacy for {c, A, b, B} and output privacy for x* is the requirement. 
The cloud customer then uses the affine transformation [l44l § III-C] 

X = (f>{z) = Mz - r , 

where M £ M"^" is a nonsingular matrix and r £ M" is a vector, both known by the customer only. The 
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equivalent problem outsourced by the customer to the cloud is given by 

minimize c^z 

subject to Az = b (8) 
Bz > , 

where the variable is z G M"* and the problem input is c = M^c, A = MA, b = b + Ar, and B = MB. 
The cloud computes the optimal solution of problem which we denote by z*. 

The sensitive inputs of problem {c, A, b, B}, cannot be recovered by a potential adversary or the cloud 
because the matrix M and the column vector r are not known to the cloud. For the same reasons, the cloud 
cannot construct the sensitive output x* by using z*. Thus, the solution procedure yields both input privacy 
and output privacy. 

The problem data (input) structure can be exploited so that change of variables can be applied to 
develop privacy preserving solution methods in multi -party environments [|26l . ETl . Il45l . H6l . 



Example 8 ( Classification, a multi-party situation f46^): Consider the following problem: 

minimize ||v||i + ||Bx||i 

subject to D(Ax - lu) + v > 1 (9) 
V > , 

where the variables are x G M", v e M™, and w G H. Here the problem parameters D = diag((ii, . . . , dm) 
with dj G { — 1, +1} is known as class matrix, A G M™^" is the feature matrix P31 . and B G M™^". The 
goal is to find the public linear classifier given by (x*)^y — it* = 0, where (x*,v*,m*) is the solution of 
problem (|9]l. Suppose A and x are partitioned as follows: 

A = [Ai, . . . , A^] , x = (xi,...,Xg) , 
where A; G 1R™^"% x^ G H"', and J2i=i — Here we have q entities and each submatrix A^ is owned 
by ith entity. The ith entity does not want to reveal its problem data Ai and the solution x* to the other 
entities j i), who might act as the adversaries. Thus, input privacy for {Ai}i=i ... g and output privacy for 
{x*}i=i .q is the requirement. 

All the entities then use the transformation ll45l § 2] x = (j){z) = B^z, where B G M™^" is a full rank 
matrix with rank(i3) = n and is horizontally partitioned as follows: B = [Bi, . . . , B^]. Here the submatrix 
Bi G ]R™^"i is known only by the ith entity. Now consider the closely related problem (see P31 equ. 2.6]) 

minimize ||v||i + ||z||i 

subject to D ^ {Y.Ui AjBj) z - Iw j + v > 1 (10) 
V > , 

with the variables are z G H™, v G H™, and u G H. Let us denote the solution by (z*,v*,it*). Provided 
B = B(B^B)^^, problem (|9]l is equivalent to problem (fTOl l. Note that in the above solution method no entity i 
reveals its sensitive input A^ and its private submatrix B^ to others. Instead, entity i needs to make public 
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only the product AjBj. As a result, no entity j {=/= i) can recover and x* = BJz* without knowing B^, 
i.e., the input privacy and the output privacy is obtained. Moreover, for any new y = (yi, . . . , y^) e R", each 
entity i makes public B^y^ from which the public linear classifier is computed as follows: 

(z*)TBy -u* = (z*)T ^'^^ B,y, - = . 

As listed in Example [6l it is also possible to use nonlinear change of variables for developing 
privacy preserving solution methods. To see this let us consider the following example: 

Example 9 (Nonlinear transformation, a 2-party situation): Alice wants to outsource to an untrusted party 
the problem 

minimize Yn^iiai/xj) 

subject to Er=i/5«^f<7 (11) 
X > , 

where the variable is x. Here, the problem data are ai > 0, > 0, and 7 > 0. Suppose Alice wants input 
privacy for problem data {cki,/3i}i=i „, 7. By using the nonlinear change of variable given in Example |6l 
we have Xi — <j>i{zi) — ai exp(zi). Next Alice can obtain the equivalent problem: 

minimize Yl'Li'^M-^i) ^^^^ 

subject to X^ILi exp(2zi) < 1 , 
where the variable is z = and problem parameter = {(3iaf/'y). Now Alice can outsource 

problem ( fT2] l to the untrusted party. We can see that this solution procedure clearly yields input privacy for 

the sensitive input {{o^i, /3i}i=i,...,n, 7} . The solution x* of original problem is simply obtained by x* = 

ai exp(z*), where z* is the solution of problem (fT2] |. 



B. Transformation of Objective and Constraint Functions 

Let the original problem to be solved in a privacy preserving manner be given by (O. Suppose 

V'o '■ E)o C R — > R is monotonically increasing, with domain covering the image of /o, i.e., 

Do ^ image /q. Moreover, suppose that for i = 1, . . . , g, : Dj C R — )■ R, with Dj D image fi, 

< if and only if z < and tjj : R^ — )■ R"^ satisfies ^{z) = if and only if z = 0. 

The equivalent problem is clearly given by 

minimize ^/'o(/o(x)) 

subject to ^i(/.(x)) < 0, z = 1, . . . , g (13) 
V^(Cx-d) = , 
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where the variable is x G R". The optimal solution of problem (fT3l) is identical to that of 
problem (O. The optimal value of problem (O, p*, and that of problem (fT3l) . g*, are related by 

Mp'') = q*- (14) 

Privacy Properties 

With carefully chosen functions tpi, and ip, any solution method for solving the transformed 
problem would yield input privacy for the input of problem ^ via the function compositions 
/i(x) = ipiifiix)) , dom = {x G dom fi \ /,(x) G dom tpi} , i = 0, . . . , g , 
/i,(x) = V'lCx - d) , dom /li = . 

Note that the optimal value p* is not available to a potential adversary, though the optimal 
solution is. As a result, output privacy is attained for the optimal value, see (fT4)) . In the sequel, 
we give some examples to highlight these ideas. 

Example 10 (Scaling): This idea is already presented in Example |5] Scaling is used in part to develop 
privacy preserving solution methods in references such as H, ll25l . Il30l . OTI . Il43l . ||44]| . Generally speaking, 
here all the functions ^/'i, i = 0, . . . , g and i/i are linear (see dTSI)). i.e., 

^i(z) = aiZ , i 0, . . . , g and V'(z) = Bz , (15) 
where {ai}i=o,....q positive scalars and B e IR^^p is a diagonal matrix with nonzero diagonal entries, 
which are unknown to any potential adversary. Specifically, the generalized permutation matrix used in ifsTl 
§ 4.1], 1 5 p. 69], the scalar 7 used in [44 III-B-3], and the positive diagonal matrix S used in ll43l § III] are 
identical to some scaling of the form ( fTsT l. 



Example 11 (Least-squares problem, a multi-party situaiton): Let us consider the basic but important prob- 
lem 

minimize jjAx — b||2 , (16) 

where the variable is x e M" and the problem data A G R™^" with rank(A) — n and b e M™. 
Suppose A and b are partitioned as follows: 



A = 



Al...Af\^ , b = (bi,...,b,) , 



9 

where A; e H™'^", hi e R™', X]i=i ™i — ™' ^^'^ rrii > n for all i = 1, . . . ,q. The submatrices A^, hi 
are owned by ith entity and altogether q entities, who want the cloud to solve problem ( fTSI l for them. The ith 
entity does not want to reveal its problem data A^ and b; to the other entities j i), and the cloud who 
might act as the adversaries. Thus, input privacy for {A^, bi}i=i ... g is the requirement. 



October 12, 2012 



DRAFT 



14 



The idea is to simply use this objective transformation ipo{z) = z^, to yield 

minimizex ||Ax — b||2 = x^A'^Ax - 2b'^Ax + b'^b 
and to let the cloud solve the resulting equivalent problem above. Clearly, we have A^A — X]?^! AJA^, 
b^A = bjAi, and b^b = bjb,. Thus, the problem can be outsourced by simply each entity i 

making public only the products of matrices {AJA^, bjA^, bjbi}. Note that nobody can construct the sensitive 
inputs of entity i {A;, bi} by knowing the public matrix products mentioned above. This means that the solution 
method for solving problem JTSI) yields input privacy for {A^, bj}i=i 



Example 12 (Horizontally Pariitioned Linear Programs, a multi-party situation i l2&l/ )." The method proposed 
by the authors in ll28l is built on the following equality constraint transformation: 

^(z) = Bz , (17) 
where the B e 11™^^ with m > p and rank(B) = p. Thus, the following equivalence holds: 

Ax - b = 4^ ip{Ax - b) 4^ B(Ax - b) = , (18) 
where A G H?"^" and b G tW. We can easily show ( fTSl l by using the pseudo-inverse of B, which is given 
by B^ = (B^B)^^B^. In reference 1281 . the authors exploit the partitioning of matrices A, b together with 
carefully chosen partitioned matrix B to develop a solution method that yields input privacy for sensitive 
input (see ll28l § 2] for details). 

One can readily apply hybrid variants of the transformation via change of variables (see § IIII-AI) 
and transformation of objective and constraints (see § IIII-BI) . Such hybrid variants can be found in 
the references [|25l . Il30l . Il43l . Il44ll . For example (roughly speaking), in [|43l § III], the authors 
have first used the change of variable x = Qz — Qr, where Q is a generalized permutation 
matrix [|3T1 § 4.1]. Then, for some constraints in the resulting equivalent problem, they have 
applied scaling by using Q^^ and a positive diagonal matrix S. 

IV. Privacy Preserving in Decomposition Methods 

In this section, we highlight important aspects of decomposition methods for designing privacy 
preserving solution algorithms. Although these techniques have been heavily used in the context 
of parallel and distributed optimization (see [HI, [|32l . [[33l . [|47l . [|48l and the references therein), 
we remark that they have not been investigated in the context of privacy preserving optimization. 
There are appealing intrinsic privacy preserving properties of these methods, as opposed to 
extrinsically acquired privacy in transformation based methods (see § Hill) and cryptographic 
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TABLE I 



Decomposition structure: 3 entities, A, B, and C with 3 nets 



A 



b 



B 



yi=ai, y2 = (bl,b2,&3), y3 = (C1,C2,C3) 

p = 7, y = (yi,y2,y3) = (ai, 6i, 62, 63, ci, C2, C3) e R'^ 

coupling: net 1 (zi): ai — bi — c\ , net 2 (2:2): &2 = C2 , net 3 (2:3): 63 = C3 

3 nets ^ z — (zi, Z2, 23) G ]R^ 



c 




y = Ez = [e7 



El = 1 



T Tl 

E2 E3 z , where 



10 10 

j , E2 = 10 , E3 = 10 
1 1 



methods. In the sequel, we present concisely the theory associated with the classical techniques of 
primal and dual decomposition [[T]|, [[32ll . Moreover, we highlight their use in privacy preserving 
optimization by a number of examples. Note that the solution methods based on decomposition 
relies heavily on the problem structure. Key techniques such as introduction of new variables, 
duality (see [|39. § 4]) are often useful in such situations to facilitate decomposition as we will 
see in the sequel. 

A. Primal Decomposition 

Suppose there are K entities and the entity i has private variables Xj G ]R"% shared or 
public variables G R^' private objective function fi : IR"' — RP\ and private constraint set 
Ci C ]R"» X ]RP\ The entities are coupled through public variables that require various scalar 
components of those to be equal. Such coupling or complications can be graphically shown 
by adding a hyper edge (or a net) that connects the entities for any scalar equality constraint 
among them. Let be the number of nets in the system and z = (zi, . . . , zn) G R^ be the 
common values of the public variables on the nets. For instance, the decomposition structure 
shown in Table |I] has 3 nets and z = (2:1,^2, ^3)- We denote by y G R*' the concatenated 
public vectors, where y = (yi, . . . ,yA') and p = J2f=iPi the total number of scalar public 
variables (see Table HI). By this notation, we let y = Ez, where 




otherwise . 



y\i IS in net j 



(19) 
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We introduce a partition of E as [Ej ■ ■ ■ E]^]^, where Ej G R^'^^ denotes the block of 
E associated with the entity i, so that = EjZ. The problem we consider is of the form 

minimize Ylf=i M^i^ Yi) 
subject to (xi, Yi) eCi, i = 1, K 
= EiZ, i = l,...,K 

ze Z , 

where the variables are {(xj,yj)}j=i_ x and z and Z is a publicly known convex set that 
enforces other potential constraints on public variables {'yi}i=i....,K- We refer z as net variables. 

Primal decomposition techniques allows us to solve the global problem (|20|) by coordinating 
K subproblems, namely one for each entity. Specifically, each entity i can solve his subproblem, 
which consists of his own private inputs/outputs (e.g., fi,Ci, Xj) together with the public variables 
Yi. The coordination performed via some message exchanges among neighbors. It is interesting 
to note that, though these messages are dependent on problem input, they are often intrinsically 
privacy preserving. In other words, the exchanged messages can be such that the input {fi,Ci} 
and output {xj} of entity i is not revealed to other entities j ^ i. To see this, let us first focus 
to the derivation of the solution method, where we give a concise discussion. 

Algorithm Development 

If we fix the net variables z (thus, public variables = EjZ is fixed), then problem (|20l) 

becomes separable into K subproblems and subproblem for entity i is given by 

minimize /i(xi,yi) ^^^^ 

subject to (xj, Yi) G C,; , 
where the variable is x,;. We denote by (/>(yi) the optimal value of problem (|2T]) . Now note that 

the original problem (l20l) is equivalent to the following, which we refer to as the master problem 

minimize 0(z) = 0i(Eiz) ^^^^ 
subject to z G Z , 

where the variable is z. The coordination among the entities resides in the method that is used to 
solve the master problem (|22l) . Projected subgradient method is a well known algorithm for such 
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a solution, see ||49l for details. In particular, when applying projected subgradient method 
to solve problem (|22l) . it is required to compute a subgradient P of at a specified z. One such 
specific subgradient is 

h = E£iEjh,(E,z) , (23) 

where hj(yj) G IR^' is a subgradient of (pi at yj. This subgradient is obtained for free after solving 
the subproblem (|2T1) . Note that hj is the message, entity i should send to his neighboring entities. 
The main algorithm is as follows: 

Algorithm 1: Primal Decomposition 

Given the initial net variables z and iteration index k = 1. 
repeat 

1. Entity i sets := E^z, solves subproblem (l2Tl l. and returns subgradient gj , i = 1 . . . , K . 

2. Update the subgradient from ( |23] l. i.e., h:= ^J^i- 

3. Update the net variables as, z:= H^: (z ~ afch), where Ilz{-) denotes the projection onto Z. Set 
k:= k+1. 

In this algorithm, ak is an appropriate step size. Algorithm [7] converges to x* = (x*, . . . , x^), 
y* = (y^, . . . , y^), z*, an optimal solution and p*, the optimal value of problem (|20l) , respectively 
(see ll32ll . [|49ll for details). It is worth noting that each step above is performed in a 
parallelized manner. This is clearly visible in the case of step 1. In the case of step 2 and 
3, each one of them boils down into a communication between neighboring entities, which 
share a common net. There is no need for any entity i to communicate with entity j, if they 
do not have a net in common. In the sequel, we establish some appealing privacy preserving 
properties of primal decomposition based solution methods. 

Privacy Properties 

There are interesting intrinsic privacy preserving properties in primal decomposition based 
methods. They can address cases which are not handled by transformation based methods (§ HIH). 
first, it is possible to yield input privacy for the input {/j, Cj}j=i_,,,_i^, and output privacy for the 

^Due to page limitations and to avoid information extraneous to the main focus of the paper, we do not go into details here. 
However, we refer the reader to (T|, 1501 for details. 
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output {{x*}j=i . This is because no entity i is required to reveal his problem data /j, Cj, 

as well as the current solution Xj to any other entity j ^ i (see step 1 of AlgorithmUl- However, 
to compute the subgradient g of the master problem's objective function, entity i should reveal hj 
(see step 1-2 of Algorithm\l\. That is the subgradients {hj}j=i ... x are the means of coordination 
between the entities. It is worth emphasizing that these subgradients do not reveal the private 
information {fi,Ci]i=i^,„^K, and {xj}j=i ... to each other in very many cases. Moreover, note 
that entity i requires coordinating or communicating only with its neighbors, who have a common 
net, in order to carry out the algorithm. As a result, there are inherent boundaries beyond which 
the information (e.g., subgradient hj) is not penetrating through multi-party environments. Let 
us give now some examples to convey these appealing aspects. 

Example 13 (Quadratic minimization, a 2-party environment): Suppose Alice and Bob wants to solve the 
following problem: 

minimize xjBiXi + X2B2X2 + 2z^(Ci + C2)z 
subject to Aixi < z 

(24) 

A2X2 < z 

xi>0, xi>0, z>0, 

where the variables are z G and x^ G IR"% i — 1,2. The problem data are A,; G 11^*^"% and positive 
semidefinite matrices G ]R"»^"» and G IR^^^, where i — 1,2. Suppose Alice wants input privacy 
for problem data {Ai,Bi,Ci} and Bob wants input privacy for problem data {A2,B2,C2}. It is worth 
pointing that the transformation based methods (see S HIll i are not applied here. To apply primal decomposition 
techniques, we first pose this problem in the form of ( |20] i: 

minimize x^BiXi + X2B2X2 + Yi Ciyi + yJC2y2 
subject to AiXi < yi 

A2X2 < y2 (25) 
yi = y2 z 

xi>0, xi>0, z>0, 

where the private variables are x^ G IR"\ public variables are y^ G H'^, i = 1, 2, and net variable is z G M''. 
For fixed z, problem (l25T l is decoupled as 

minimize xJB^x^ + y^C^y,; 

subject to AiX,; < y^ (26) 
xi > , 

where the variable is x,; G M"' (let i — \ corresponds to Alice and i = 2 coiTesponds to Bob). By using the 

usual notation 0(yi) to denote the optimal value of problem above, we can show that 

h,(y,) =2Qy, -A, , (27) 
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where Aj is an optimal dual variable associated with the first constraint of problem ( |26l ). Thus, Alice makes 
public hi and Bob makes public h2 and they communicate to perform Algorithm Q] Note that Bob cannot 
recover the input {Ai, Bi, Ci} of Alice because Ai is known by Alice only and hi is a transformed variant of 
Ci. Same arguments hold for Alice as well. Thus the solution method yields input privacy for {A^, B^, C^ji^i 2- 
Moreover, we can easily see that the algorithm yields output privacy for {x|,X2}, the solution, and for p*, 
the optimal value of the original problem (l24l l: p* is not known to anyone. 



Example 14 (Linear programming with coupled constraints, a multi-party environment): Suppose K enti- 
ties wants to solve the following problem: 

minimize X^ili 

subject to Ya=i AjXi < X][Li bi (28) 
Xi > , i = l,...,K , 

where the variables are x; e i — 1,...,K. The problem data are A^ G Rp^"' and e M"', and 

e H^*, where i — 1,...,K. Suppose entity i wants input privacy for problem data {Ai,hi,Ci}. We 
emphasize again that the transformation based method (see ij HIH i are not applied here because of {hi}i^i x. 
But primal decomposition methods are applied gracefully. To show this, let us first reformulate the problem 
equivalently as follows: 

. . . sr^K T 
minimize Z^,-i Xi 

subject to A,;Xi - < , Y.f=i Yi = (29) 
Xi > , i ^ 1, . . . ,K , 

where the private variables are x, G M"' and the public variables are G H'', i = 1, . . . ,K. In the problem 

above, we have not explicitly written out the net variables z = (yi,...,yx) G and the constraint 

[Ii • • • = 0, where 1^ G H^'^^' is the identity matrix. This is to avoid unnecessary notations. For fixed 

{yi}i=i,...,K, problem ( l29l l is decoupled as 

minimize cjxi 

(30) 

subject to AiXi — b,; < y^ , xi > , 
where the variable is x^ G M"*. With usual notations, we obtain the subgradients as follows: 

h,;(y.) = -[Oi, • • • , I,;, • • • , OkVK , h e W'P , 0, G , J^t, (31) 

where A^ G MP is an optimal dual variable associated with the constraint A^x; — b; < y^ of problem ( l26l l. 

Thus, each entity makes public its h^ and they communicate to perform Algorithm\J] Note that no entity j i 

can recover the input of entity i because Ai has no information of {Ai,hi,Ci}. Here, the step 2 and 3 of 

Algorithm [7] is simply given by 

h:= (h,;,...,hK) , z:= [[Ii--.lA'](z-afeh)]+ , (32) 

where I,; G IRp^p is the identity matrix and [ y ]+ is the projection of y G IR^ onto set the cone 

of nonnegative, real p-vectors. We note that solution method yields input privacy for {A^, b^, c^ji^i as 

required. We also note that the algorithm yields output privacy for the solution of original problem ( |48] | (i.e., 

{x*}i=i ...x) and for the optimal value (i.e., p*), because none can compute it. 
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B. Dual Decomposition 

Dual decomposition techniques provide another mechanism for solving problems in a par- 
allelized manner. As the name suggests, the machinery behind the dual decomposition is the 
duality |[391 § 5]. They can gracefully handle cases where either the transforation based meth- 
ods (§ Uni) or the primal decomposition based methods (§ IIV-AI) are not applied. In the sequel, 
we first present concisely the dual decomposition algorithm derivation. Then we discuss the 
privacy preserving properties of this method followed by some examples. 

Here, we use the same problem (|20|) that we considered in § IIV-AI We start by writing the 
partial Lagrangian of problem (l20l) 

L(x, y, z, A) = E.=i /.(x., y.) + AT(y - Ez) (33) 

= E£i (/.(x., y^) + A[y,) - A^Ez , (34) 

where A = (Ai, . . . , Xk) E W is the Lagrange multiplier associated with y = Ez, and Aj is 

the part of A associated with entity i. The dual function is obtained by minimizing L(x, y, z, A) 

over (x, y,z). For simplifying the presentation, suppose Z = M^. Thus, the minimization with 

respect to z indicates that E^A = (otherwise, the dual function is unbounded below), which 

acts as a constraint on dual variables A. Moreover, because the Lagrangian (|34|) is decoupled, the 

minimization with respect to (x, y) can be carried out independently, i.e., each entity performs 

minimization with respect to (xj,yi). In particular, entity i solves the problem 

minimize /^(xi, y^) + Ajy^ ^^^^ 

subject to (xj, Yi) eCi , 
where the variable is (xj,yj). We denote by (x*,y*) the solution and by gi{Xi) the optimal 



value of problem (|35|) . Then, the dual problem can be formally expressed as 

maximize g{X) = J2f=i9ii>^i) 

(36) 

subject to E'A = , 

where the variable is A. Moreover, we can easily show that — y* = — (y*, . . . .Y^k) ^ subgra- 
dient of — 5'(A) at A. We use projected subgradient method to solve the dual problem (l36l) . The 
algorithm is as follows: 
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Algorithm 2: Dual Decomposition 

Given an initial value of A such that E^A = 0. Set iteration index k = 1. 
repeat 

1. Entity i extracts subvector i of A, i.e., Ai and solves problem dSSl ). We denote by (x^, the solution, 
i^l...,K . 

2. Update the subgradient — y = — (yi, . . . , y^) . 

3. Update the A as, A:= Hg (A + a^y), where ^s{') denotes the projection onto 5 = {s [ E^s — 0}. 
Set k:=k+l. 

Here ak is an appropriate step size. As in the case of primal decomposition algorithm, each 
step of the above algorithm are performed in a parallelized manner as well. Provided some 
conditions hold true (e.g., strict convexity of finiteness of /j), A/^onY/jm |2] converges to an 
optimal solution and the optimal value of problem (|20l) . Due to page limitations, we refer the 
reader to [[II, [[32l . [|49l for details. Let us now discuss the privacy preserving properties of 
distributed optimization methods based on dual decomposition. 

Privacy Properties 

Privacy preserving properties of dual decomposition method are very similar to those of 
primal decomposition based solution methods. For example, the method yields input privacy 
for the input {fi,Ci}i=i^,,,^K, and output privacy for the output {{x*}j=i^...^i^,p*} (see step 1 of 
Algorithm^. In particular, no entity i is required to reveal her problem data /«, Q, as well as the 
current solution Xj to any other entity j ^ i. In order to perform step 2-3 of the algorithm, entity i 
should however expose the subgradient y^, which is indeed the global variable solution at the 
current iteration. These subgradients do not expose the private information {fi,Ci}i=i^,,,^K, and 
{xj}j=i ... i<- to each other. As for the primal decomposition method, entity i requires coordinating 
only with its neighbors, and therefore lesser the information (e.g., subgradient y,) penetration 
through the multi-party environment. An example is provided next to highlight these appealing 
privacy preserving aspects of the dual decomposition method, which is not handled by either 
the transforation based methods (§ [nil) or the primal decomposition based methods (§ IIV-AI) . 
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following problem that is to be solved by K entities: 

EK Ttd t T 
^^^x'BiX + X;,=ic/x ^^^^ 

subject to AjX <bi, i = 1, . . . , K , 
where the variable is x € M". The problem data are C; G MP, A; G M'^^, and positive definite matrices G 
jpj^pxp Suppose entity i wants input privacy for problem data {Ai,Bi,Ci}i=i i^. The dual decomposition 
method gracefully handles the problem. To see this, let us first equivalently reformulate the problem in the 
form (l20l l. i.e., 

minimize YlZi y^B.y, + cjyi 

subject to AiYi <hi , i ^ I, . . . ,K (38) 
yi = z , i = l,...,K , 

where the variables are z and y^, i = 1, . . . , K. Next, each entity i solves his subproblem (step 1 of Algorithm^: 

minimize yjB.y, + cjy, + Xjy, ^^^^ 
subject to A^yi < , 

where the variable is y^. Only the solution y* is made public by entity i. Note that y* contain no information 
about {Ai,Bi,Ci}, the private input of the entity. Then all entities communicate to perform step 2-3 of the 
algorithm. In particular, step 2-3 can be expressed compactly as 

A:=A + afe(y-z) or Xi:= X.^ + ak{yi - z^) ,i ^ 1, . . . , K , (40) 
where z — (z^, . . . ,zk) G and z.^ ~ i^/K) X^jLi yi ^ Note that this solution method yields input 

privacy for {A;, B^, Ci}i=i x as required. We also note that the algorithm yields output privacy for p*, the 
optimal value, because it is not computable during or after the algorithm terminates. 

In addition to the classic methods described above (see § ITVl) . there are other related solution 
methods as well, e.g., the state-of-the art alternating direction method of multipliers [[331 . In the 
sequel, we discuss basic theoretical backgrounds for these methods together with some examples 
to show their relevance in the context of privacy preserving optimization. 

V. Alternating Direction Method of Multipliers (ADMM) 

As we pointed out in § |IV-B[ to yield convergence, the dual decomposition method relies on 
assumptions such as strict convexity of /jS and finiteness of /jS. ADMM provides an elegant 
way to exclude such assumptions and thus brings robustness to the dual decomposition method, 
while still preserving the decomposability of problem (|20l ). Let us next discuss the basic theory 
behind ADMM. We refer the reader to [33J for details. 

The problem formulation is slightly changed, i.e., instead of (general) problem (|20l) . we 
consider a simple variant that is sufficient to highlight the steps of algorithm development. 
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minimize /i(y) + /2(z) ^^^^ 

subject to Ay + Bz = c , 
where the variables are y G IR", z e W and the problem data is A G R'^^", B G IR''^^', and 

c G R''. Let us now consider a related problem obtained by adding a quadratic penalty term to 

the objective function of problem (|4TI) . i.e., 

minimize /i(y) + /2(z) + (p/2)||Ay + Bz - c 

subject to Ay + Bz = c , 
where the variables are (y, z) and p > is called penalty parameter. We can easily show 

that problem (|4T]) is equivalent to problem (l42l) . Note that the ADMM is associated with the 

equivalent problem above. By following a similar approach as in § IIV-BI we obtain the partial 

Lagrangian of problem (|42l) as 

L(y, z, A) = /i(y) + ^(z) - A^(Ay + Bz - c) + (p/2)||Ay + Bz - c||2 , (43) 

where A G R*^ is the Lagrange multiplier associated with Ay + Bz = c. The minimization of 

the Lagrangian is performed in two steps, as opposed to one step in dual approach. In particular, 

we have 1) y-minimization, 2) z-minimization. Next, the dual variable update is performed. The 

algorithm continues in an iterative manner as follows. 



Algorithm 3: ADMM (unsealed) 



Given an initial variables A'-^-', z'^^ and set fc = 0. 
repeat 

1. y(^-+i) argmin/i(y)-AT(Ay + Bz(^-)-c) + (p/2)||Ay + Bz(*)-c||2 

y 

2. zf'^+i) argmin/2(z)-A^(Ay('^+i'+Bz-c) + (p/2)||Ay('^'+i)+Bz-c||| 

3. aC^+i) := A*'"' + p(Ay(^+i) + Bz^'^+i) - c) . k:=k + l . 

The convergence of the algorithm above is guaranteed with very little assumptions (e.g., /i, /2 
are convex, closed, proper), see [|33l § 3.2] for details. We skip those due to page limitations. 
By defining the residual r = Ay + Bz — c, a slightly different form of ADMM algorithm can 
also be obtained [33, § 3.1.1], which is given by 
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Algorithm 4: ADMM (scaled) 

Given an initial variables /^t^°^, z^^^ and set fc = 0. 
repeat 

1. yC^+i) :== argmin/i(y) + (p/2)||Ay + Bz(''') -c + /x('=)||^ 

y 

2. zf'^+i) argmin/2(z) + (p/2)||Ay('=+i) +Bz-c + /x('=)||^ 

3. /xC^+i) := /xf'^^) + AyC^'+i) + BzC^+i) - c . fc:=:fc + l . 

Privacy Properties 

To give the general flavor of those properties, suppose /i, A in problem (|4TI) are owned by 
entity 1 and /2, B are owned by entity 2 and c is public. In step 1 of Algorithm\3\{or^, entity 2 
should make public the product Bz^''\ Because entity 1 does not have access to z^''\ he cannot 
construct entity 2's private data matrix B. Similar arguments holds when entity 2 attempts to 
construct entity I's private data matrix A as well, see step 2 of the algorithm. Particularized to 
problem (|20|) . all the privacy preserving properties of dual decomposition method are achieved 
by ADMM as well. For instance, the method yield input privacy for the input {/«, Cj}j=i ... 
and output privacy for the output {{x*}j=i Note that the penalty term here would be 

(p/2) I |y - Ez| |2 = (p/2) Zti I |y. - E.z| |2 , (44) 
which is separable among the entities. As a result, the step 1 of Algorithm \3\ (or |4]) would 
be separable and allows each entity i to solve its' subproblems without revealing to any other 
entity j ^ i the problem data fi,Ci and the current solution. The coordination in step 2-3 
of the algorithm is through the global variables {yi}i=i,...,K, which do not expose the private 
information {fi,Ci}i=i^,,,^K, and {xj}j=i to each other. 

It is important to note that there is a number of other interesting problems where transfor- 
mation (§ Inil), primal (§ IIV-AI) . and dual (§ IIV-BI) based methods do not applies, but ADMM 
method gracefully applied to solve those in a privacy preserving manner (e.g., problems with 
functions that takes on oo). In the sequel, we provide some example to give these flavors to the 
reader. 



October 12, 2012 



DRAFT 



25 



Example 16 (Intersection of two closed nonempty convex sets): Suppose Alice owns a private set A C IR"- 

and Bob owns a private set B C H". They want to find a point x such that x G ^ n S, without revealing the 

private sets. We denote by /i the indicator function of set A, and by /2 the indicator function of set B. Thus, 

the problem can be posed as n , \ ^ , \ ,ac\ 

minimize /i(z) + /2(z), (45) 

where the variable is z G R". ADMM method can readily be applied solve this problem in a privacy preserving 
manner. Consider the equivalent problem 

minimize /i(y)+/2(z) 

(46) 

subject to y = z , 

where the variables are y G M" and z G H". Thus, given z and /x, step 1 of AZgonf/im |?] simply reduces to 

returning the solution y* of problem ... / ,„mi , mi9 

minimize (p/2)||y - (z - ^^^^ 

subject to y G ^ , 

with variable y. The solution y* is simply n_4(z — /x), where denotes the projection on to A. Given y 
and step 2 of Algorithm^is identically solved by z* = nB(y — fi), where Ilg denotes the projection on 
to B. finally, step 3 reduces to := /x^*^' + y* ^ z*. Clearly, the solution method yields input privacy 

for {A, B}. 



Example 17 (Linear programming with partitioned data, a multi-party environment): Suppose K entities 

wants to solve the following problem: ^ 

minimize V , cjz 

K K (48) 

subject to Y.i=i Ajz < Y.i=i , 
where the variable is z G M". The problem data are G IR^'^" and G M", and G W, where i = 

1, . . . ,K. Suppose entity i wants input privacy for problem data {A^, bj, Ci]. Note that transformation (§ HIH i. 

primal decomposition (^ IIV-AI ). and dual decomposition (ij llV-BI ) based methods do not applies here. However, 

as we will explain, ADMM is readily applied. To show this, let us first reformulate problem ( l48T l as follows: 

minimize Y^'Li ^Jvi 

subject to A^y; - b^ ^ z.^ , i^l,...,K ^^^^ 
Yi ZA'+i , i = l,...,K 

where the variables are {yj G IR"}j=i,...,/f , {zj G ^^}i=i,...,K, and z/^+i G M". We denote by Aj G TRP 

and /Xj G R" the (scaled) Lagrange multipliers associated with A^y^ — b^ and y^ — zk+i, respectively. Thus, 

given {z Jj=i...,A'+i, {>^i}i,....K, and {/xji,...,^, step 1 of A/goWf/zml?] returns y* (yj, . . ■ ,y*j^), where y* 

is the solution of problem 

minimize cjy, + (p/2)||Aiy, - b^ - z,; + Ai||| + (p/2)||yi - zk+i + mJH , (50) 
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with the variable y^. Given y, A, and fi, step 2 of Algorithm ^retains (z|, . . . ,zj,^j^), the solution of 

minimize {p/2)J2f^^ \\z, - (A.y^ - b, + A,)||^ + {p/2)J2f^^ \\zk+i - iVt + ^J'^)\\^ 
subject to X]i=i < , 

with the variable (zi, . . . , zk+i)- Because we want to see the relation of data decomposition in each prob- 
lem (l50l l. as well as (fsTl i. here we do not attempt to find any closed form solutions to those problems, finally, 
step 3 of Algorithm^ i.e., the (scaled) dual variable update is given by a'''"'^^'' — a''''' + (A^y* — — z*) 
and Atf = ^ + (y. - zs), t^l...,K. 

Note that problem (fSOl l is solved by entity i. Therefore, the problem data {Ai,hi,Ci} of entity i is not 
revealed to anyone during the step 1. Moreover, in step 3, entity i should reveal m, — A^y^ — b^ + A^ to all the 
other entities. However, the input {A,, bi, c^} is hidden because no other entity knows changing (in algorithm 
iterations) A and as a result, no one can estimate A^ and b^. Therefore input privacy is obtained. Moreover, 
we see that the solution method yields output privacy for the optimal value of the original problem ( l48T l. 



VI. Discussion 

In this section we compare and summarize the main characteristics of privacy preserving 
methods for distributed optimization, including cryptographic techniques, transformation based 
methods, decomposition based methods, and ADMM. The main focus of this comparison is to 
identify the advantages and the disadvantages of each solution method, thus defining some high 
level guidelines for possible application in terms of performance indicators such as efficiency, 
scalability, etc. 

Table HI] summarizes the results of our comparison. We emphasize that this table is to a give 
high level comparison on privacy preserving approaches and is not meant to be comprehensive. 
Each row of the table is by its own important and can be discussed extensively. However, in 
the sequel, we highlight key related issues. In the case of first four indicators, we use an integer- 
scale of 1—5, where, 1 stands for 'How" and 5 stands for ''high" and 2—4 stand for in between 
cases. 

Here, the term ' 'efficient" can mean the effective implementation of the protocols in general. 
Particularized to optimization based methods, efficiency further can mean the fast convergence 
of the associated iterative algorithms. For many non-trivial problems, cryptographic treatments 
require very large circuit representations, resulting explosions in associated computation due to 
integers involving thousands of bits [5, § 2.3.2] [29] . However, many optimization methods use 



October 12, 2012 



DRAFT 



27 



TABLE 11 

Privacy preserving techniques comparison 



Performance Indicators 


Cryptography 


Transformation 


Decomposition 


ADMM 


1. Numerical efficiency 


1 


5 


2-3 


4 


2. Protocol complexity 


5 


1 


1 


1 


3. Scalability 


1 


2-4 


5 


5 


4. Privacy/security level 


5 


7 


7 


7 


5. Protocol-solver dependence 


yes 


no 


no 


no 


7. Linear programming restriction 


yes 


no 


no 


no 


8. Prob. structure and constraint dependence 


no 


yes 


yes 


yes 


9. Data partitioning dependence 


no 


yes 


yes 


yes 


10. finite field restriction 


no 


yes 


yes 


yes 


1 1 . Real field restriction 


yes 


no 


no 


no 



the state-of-the interior-point solvers [|39l § 11], which can be implemented by using floating- 
point arithmetic and thus they are very effective compared to cryptographic methods. These 
properties are directly reflected in the case of protocol complexity as well. In terms of scalability, 
decomposition methods, and ADMM are preferred compared to others. This is because the 
mechanism associated allows them to coordinate their actions using a thin protocol irrespective of 
the size of the problem. The poor scalability properties of cryptographic methods is mainly due to 
their low efficiency and high complexity, especially for relatively large problems. Transformation 
methods also have scalability issues because a transformed variant of the original problem has 
to be solved by each party involved. As a result, the computations are restricted by the size 
of the problem. There are guaranteed privacy properties in cryptographic methods jU, ll34l — 
[|38l . However, in the case of optimization based approaches, there are little investigations for 
quantifying the privacy. This is still a largely a question mark and an open issue. 

In the case of linear programming, cryptographic protocols are often collaboratively executed 
in each iteration step of the simplex algorithm [[51, [|44ll . Thus, they are inherently restricted 
to linear programs. More recently the first cryptographic method for handling interior point 
method have been proposed in [[H . Still the high computational complexity for cryptographically 
secured iterations in both the simplex and interior point methods is unavoidable. However, the 
underlying machinery for interior point methods for optimization based approaches are built on 
linear algebra, which is very efficiently implemented compared to cryptographic treatments. A 
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basic characteristic of the cryptographic-based methods is that they are carried out over a finite 
field and thus is constrained to integer domains. In contrast, optimization based approaches are 
well suited for real vector spaces, finally, note that the cryptographic methods are not much 
sensitive to the structural changes of the problem, though the protocol is required to be updated. 
However, the decomposition, and ADMM, methods are highly sensitive to the way the data are 
partitioned among different parties. 

All the distributed privacy preserving methods presented (decomposition, ADMM) in the 
paper solve the problem in an iterative manner (in the original variable domain) compared to 
cryptographic and transformation based method. Each iteration requires the entities to share 
some information about its current state. This collecting of data from several iterations might 
allow an adversary to estimate private data. This situation can appear in any of the distributed 
methods. We strongly believe that quantifying the security level in these situations as well as 
in different optimization approaches is still an open question, requiring a careful discussion of 
the algorithmic properties as well as the problem assumptions. 

VII. Conclusion 

In this paper we presented a framework for a systematic study and classification of distributed 
optimization solvers that are able to preserve the privacy among parties. A general classification 
of the privacy preserving methods was presented. We summarized the existing privacy preserving 
techniques by a more general form and we discussed in detail general decomposition structures 
in optimization and their privacy preserving properties. 

Several challenging issues still arise in this field, first, transformation and decomposition 
methods provide security guarantees that are not strong, especially when applied in small 
problems. In these cases the disguise methods that are applied do not have enough inputs 
(variables and constraints) to securely transform the problem to a new domain. Therefore, more 
research effort must be placed in establishing new methodologies that could guarantee high 
security levels even in small scale problems. Second, the efficiency of the transformation and 
the decomposition methods highly depends on the way data are partitioned among the participants 
in the optimization. We believe that research is required to define a systematic method to disclose 
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the best partitioning scenario so to reduce the vulnerabiUty of the method. We beUeve that future 

parallel and distributed systems that wish to optimally interact while preserving privacy, can gain 
a significant benefit from the methods we have surveyed. We further believe that the theory of 
per-se privacy preserving optimization is at a very early stage, an much needs to be investigated. 
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